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OBJECT DETECTION SYSTEM AND METHOD OF DETECTING OBJECT 
BACKGROUND OF THE INVENTION 

1. Field of Invention 

5 [0001] The invention relates to an object detection system and a method of detecting an 

object that detects an object by means of radar and images. 

[0002] Recendy an operation support system, for example, a collision avoidance control 
system, an adaptive cruise control system, a tracking control system and the like has been 
increasingly developed. The detection of an object (obstruction) such as a preceding vehicle is 

10 essential for the operation support system. An object detection system disclosed in 

publications JP-A-2003-84064 and JP-A-7-125567 as below includes two detection units, for 
example, a radar such as a laser radar, and a stereo camera that shoots an image. The object 
detection system performs collation between the detection results of the radar and the detected 
image. The preceding object is detected based on the aforementioned collation results. 

15 [0003] In the generally employed object detection system, each of two kinds of detection 

units detects an object that is considered as being existent In other words, the object detection 
system may occasionally detect a plurality of objects as the preceding object. In this type of 
object detection system, the collation of the detection results is performed in one-way process, 
that is, the detection result of one detection unit is collated with respect to the detection result of 

20 the other detection unit so as to identify the object. The collation in the inverted process, 

however, is not performed. This may cause the object detection system to mistakenly identify 
a plurality of objects. The thus identified pluralities of objects have to be further subjected to 
the process for narrowing into the single object by performing complicated operations. Even if 
the operation is performed, the identified objects may not be narrowed into the single object. 

25 

SUMMARY OF THE INVENTION 
[0004] It is an object of the invention to provide an object detection system and a method 
of detecting an object capable of detecting an object through a simple process with high 
accuracy. 

CONFIRMATION COPY 
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[0005] An object detection system according to the invention is provided with radar 
detection means that detects an object using radar, image detection means that detects an object 
using an image, and collating means that performs collation between a detection result of the 
radar detection means and a detection result of the image detection means. The collating 
5 means detects a combination of an object detected by the radar detection means and an object 
selected among those detected by the image detection means, which is the closest to the object 
detected by the radar detection means, detects a combination of an object detected by the image 
detection means and an object selected among those detected by the radar detection means, 
which is the closest to the object detected by the image detection means, determines whether 

10 there is a coincidence between the combination of the object detected by die radar detection 
means and the selected object as being closest thereto and the combination of the object 
detected by the image detection means and the selected object as being closest thereto, and 
determines, when there is the coincidence, that the object detected by the radar detection means 
is the same as the object detected by the image detection means. 

15 [0006] A method of detecting an object in a system according to the invention is 

provided with radar detection means that detects an object using a radar, image detection 
means that detects an object using an image, and collating means that performs collation 
between a detection result of the radar detection means and a detection result of the image 
detection means. The method comprises the steps of detecting a combination of an object 

20 detected by the radar detection means and an object selected among those detected by the 

image detection means, which is the closest to the object detected by the radar detection means, 
detecting a combination of an object detected by the image detection means and an object 
selected among those detected by the radar detection means, which is the closest to the object 
detected by the image detection means, determining whether there is a coincidence between 

25 the combination of the object detected by the radar detection means and the selected object as 
being closest thereto and the combination of the object detected by the image detection means 
and the selected object as being closest thereto, and determining, when there is the coincidence, 
that the object detected by the radar detection means is the same as the object detected by the 
image detection means 
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[0007] In the above-structured object detection system and method, the object is detected 
by both the radar detection means and the image detection means. All the detection results of 
the image detection means are collated with respect to the respective objects detected by the 
radar detection means by the collation means. The collation means then selects the object 

5 among those detected by the image detection means as being the closest to the object detected 
by the radar detection means. The object among those detected by the radar detection means is 
combined with the selected object that is the closest thereto. All the detection results of the 
radar detection means are collated with respect to the respective objects detected by the image 
detection means by the collation means. The collation means then selects the object among 

10 those detected by the radar detection means as being the closest to the object detected by the 
image detection means. The object among those detected by the image detection means is 
combined with the selected object that is the closest thereto. It is determined whether there is a 
coincidence between the combination of the object detected by the radar detection means and 
the selected object as being closest thereto and the combination of the object detected by the 

15 image detection means and the selected object as being closest thereto. When there is the 

coincidence, it is determined that the object detected by the radar detection means is the same 
as the object detected by the image detection means. Accordingly the object included in the 
coincident combinations is judged as being the one to be detected by the object detection 
system. In the object detection system, the detection results of two kinds of detection meanss 

20 are collated bilaterally so as to select only one object among those detected by one detection 
means collated with respect to the object detected by the other detection means in the 
respective collating processes. Further the object that satisfies AND condition between the 
aforementioned combinations is set as the object to be detected by the object detection system, 
resulting in high detection accuracy. In the object detection system, the object among those 

25 detected by one detection means which is closest to the object detected by the other detection 
means is selected to form the respective combinations. It is then determined whether there is a 
coincidence between the combinations. Accordingly the object to be detected by the object 
detection system may be easily identified based on detection results of the aforementioned two 



J 

WO 2005/024464 PCT/EB2004/002931 

4 " 

types of detection meanss through the simple process as described above while reducing the 
processing load. 

[0008] A millimeter-wave radar, laser radar and the like may be used as the radar 
detection means. A stereo camera may be used as the image detection means. 
5 [0009] According to the aspect of the invention, the determination whether each 

detection result of the radar detection means and the image detection means indicates the 
identical object to be detected by fee object detection system may be made with high accuracy 
using a simple collating process. 



10 BRIEF PESCRIPTTON OFTHE DRAWINGS 

[0010] The foregoing and further objects, features and advantages of the invention will 
become apparent from the following description of preferred embodiments with reference to 
the accompanying drawings, wherein like numerals are used to represent like elements and 
wherein: 

15 Fig. 1 is a view that shows a structure of an obstruction detection system as an embodiment 

according to the invention; 

Fig. 2 is an explanatory view representing the collating process between objects detected by 
a radar detection unit as a millimeter-wave radar and objects detected by an image detection 
unit as a stereo camera, which are provided in an obstruction detection system as shown in Fig. 

20 1; and 

Fig. 3 is a flowchart representing a collating process executed by the obstruction detection 
system as shown in Fig. 1. 

DPT ATT F.n DESCRIPTION OF PREFERRED EMBODIMENTS 
25 [0011] An embodiment of the object detection system according to the invention will be 

described referring to the drawings. 

[0012] In this embodiment, the object detection system according to the invention is 
applied to an obstruction detection system provided in a vehicle so as to detect an obstruction 
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that precedes the vehicle. The obstruction detection system according to this embodiment is 
provided with two detection units, that is, a millimeter-wave radar and a stereo camera. 

[0013] Referring to Fig. 1, an obstruction detection system 1 will be described. Fig. 1 
shows a structure of the obstruction detection system according to this embodiment. 

5 [0014] The obstruction detection system lis provided in a vehicle and detects an 

obstruction, that is, a vehicle or the like that precedes the vehicle. The obstruction detection 
system 1 functions in providing obstruction information to the operation support system such 
as a collision avoidance control system, an adaptive cruise control system, a tracking control 
system and the like; which requires the information with respect to the preceding obstruction. 

10 In the obstruction detection system 1 , each detection result of two detection units is collated 
through a simple process such that the obstruction that precedes the vehicle is identified with 
high accuracy. The obstruction detection system, thus, is provided with a millimeter-wave 
radar 2, a stereo camera 3, and an ECU 4 (Electronic Control Unit). The obstruction detection 
system 1 may be provided independently from the operation support system. That is, it may be 

15 structured to transmit the detected obstruction information to the operation support system. 
Alternatively the obstruction detection system may be assembled with die operation support 
system 

[0015] In this embodiment, the millimeter-wave radar 2 corresponds the radar detection 
unit, the stereo camera 3 corresponds the image detection unit, and the ECU 4 corresponds the 
20 collating unit 

[0016] Li tins embodiment, a millimetric-wave object refers to an object detected by the 
millimeter-wave radar 2, and an image object refers to an object detected by the stereo camera 
3. A fusion object refers to the object identified as the one that has been detected by the 
millimeter-wave radar 2 and die stereo camera 3 through collation between the millimetric- 
25 wave object and the image object It serves as the obstruction information supplied by the 

obstruction detection system 1. Hie independent millimetric-wave object represents the object 
that is detected by the millimeter-wave radar 2 only. In other words, the independent 
millimetric-wave object is obtained by excluding the fusion object from the millimetric-wave 
objects. The independent image object represents the object that is detected by the stereo 
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camera 3 only. In other words, the independent image object is obtained by excluding the 
fusion object from .the image objects. The respective objects exhibit the information with 
respect to the distance between the vehicle and the preceding obstruction, the relative speed of 
the obstruction with respect to the vehicle, the angle defined by the obstruction and the running 
5 direction of the vehicle (information of the lateral position) for determining the position of the 
obstruction with respect to the vehicle. 

[0017] The millimeter-wave radar 2, radar for detecting an object using a millimetric 
wave, is attached in the cento- of a front surface of the vehicle. The millimeter-wave radar 2 
scans the millimetric wave on the horizontal plane so as to be emitted from the vehicle forward, 

10 and receives the reflected millimetric wave. The millimeter-wave radar 2 measures the time 
period elapsing from die emission to the receipt of the millimetric wave such that the distance 
from the front end of the vehicle to the preceding object is calculated. The millimeter-wave 
radar 2 further calculates a relative speed of the vehicle with respect to that of the preceding 
object using Doppler Effect The millimeter-wave radar 2 detects the direction of the 

1 5 millimetric wave that reflects most intensely, based on which an angle defined by the traveling 
direction of the vehicle and that of the preceding object is calculated. The millimeter-wave 
radar 2 is capable of detecting the object upon receipt of the reflecting millimetric wave. At 
every receipt of the reflecting millimetric wave, therefore, one millimetric object is obtained. 
The millimeter-wave radar 2 serves to calculate the distance, the relative speed, and the angle. 

20 However, the ECU 4 may be structured to calculate those values based on detection results of 
the millimeter-wave radar 2. 

[0018] The millimeter-wave radar 2 is capable of detecting the distance and the relative 
speed with relatively higher accuracy, but detecting the angle with relatively lower accuracy. 
As the millimeter-wave radar 2 calculates the distance based on the time elapsing from i 

25 emission of the millimetric wave to reflection thereof, the accuracy in the calculated distance is 
relatively higher. As the relative speed is calculated using Doppler effect, the resultant value of 
the relative speed exhibits high accuracy. The millimeter-wave radar 2 fails to identify the 
point at which the millimetric wave reflects most intensely in the width direction of the object 
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As a result, the position in the width direction (lateral position) is likely to fluctuate, reducing 
accuracy in the angle. 

[0019] The stereo camera 3 includes two sets of CCD cameras (not shown), which are 
arranged apart at a distance of approximately several 10 cm in the horizontal direction. The 
5 . stereo camera 3 is also attached in the center of the front surface of the vehicle. The stereo 
camera 3 transmits the respective image data shot by those two CCD cameras to an image 
processing portion (not shown). The image processing portion may be assembled with the 
stereo camera 3 or formed within the ECU 4. 

[0020] The image processing portion identifies the object based on the respective image 
10 data, and obtains the positional information with respect to the object The stereo camera 3 is 
capable of detecting the object when the object is identified based on two image data. At every 
identification of the object, one image object is obtained. 

The image processing portion calculates fee distance from the front end of the vehicle to the 
preceding object by triangulation using the difference in views of the object between two 

15 image data. The image processing portion calculates the relative speed based on the change in 
the calculated distance as an elapse of time. The image processing portion detects both ends of 
the detected object in the width direction such that each angle defined by the traveling direction 
of the vehicle and the respective ends of the object is calculated. Accordingly the lateral 
positional information of the image object includes two types of angle information with respect 

20 to both ends of the object in the width direction. : 

[0021] Each detection result of the distance and the relative speed of the stereo camera 3 
exhibits relatively lower accuracy but the detection result of the angle exhibits relatively higher 
accuracy. As both ends of the object in the width direction can be detected with high accuracy 
based on the left and right image data, the detection result with respect to the angle may exhibit 

25 high accuracy. However, as the image data are supplied from left and right CCD cameras 
apart at several 10 cms, the triangulation is performed at substantially acute angle for 
calculating the distance. Accordingly, each accuracy of the distance and the relative speed is 
reduced. 
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[0022] The ECU 4, that is, electronic control unit, includes, a CPU (Central Processing 
Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The 
ECU 4 is connected with a millimeter-wave radar 2 and a stereo camera 3. The ECU 4 
receives the millimetric-wave object from the millimeter-wave radar 2, and the image object 
5 from the stereo camera 3. Then the ECU 4 performs collation between the millimetric-wave 
object and the image object so as to obtain obstruction information, for example, fusion object, 
independent millimetric-wave object, independent image object. The ECU 4 receives the 
image data from the stereo camera 3, based on which the image object is obtained. 

[0023] Referring to Fig. 2, the collating process executed by the ECU 4 will be described. 

10 Fig. 2 is a view that represents the process for performing collation between the millimetric- 
wave objects and the image objects. The example shown in Fig. 2 indicates five millimetric- 
wave objects Ml to M5 detected by the millimeter-wave radar 2, and six image objects II to 16 
detected by the stereo camera 3. 

[0024] If the millimetric-wave objects by the number of n.m are detected by the 

15 millimeter-wave radar 2, the ECU 4 fetches each of the millimetric-wave objects one by one. 
The ECU 4 then collates each of the image objects by the number of n_i with respect to the 
fetched millimetric-wave object sequentially such that the image object closest to the fetched 
millimetric-wave object is selected. La this case, the distance between the vehicle and the 
object and the angle defined by the vehicle and die object are collated, and, if required, the 

20 relative speed of the object with respect to the vehicle may also be collated. In this example, 

the difference between the distance from the fetched millimetric-wave object to the vehicle and 
the distance from the closest image object to the vehicle is set as a threshold value (several 
meters) in accordance with die accuracy of the millimeter-wave radar 2 for the purpose of 
performing the distance collation. If the aforementioned difference in the distance is equal to 

25 or larger than the threshold value, the closest image object cannot be selected. In this example, 
the difference between the angle defined by the fetched millimetric object and the traveling 
direction of the vehicle and the angle defined by the closest image object and the traveling 
direction of the vehicle is set as a threshold value (several degrees) in accordance with the 
millimeter-wave radar 2 for the purpose of performing the angle collation. If the 
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aforementioned difference in the angle is equal to or larger than the threshold value, the closest 
image object cannot be selected. If the closest image object is selected, the ECU 4 stores a pair 
of the fetched millimetric-wave object and the selected image object closest as a millimetric- 
wave base pair. The ECU 4 repeats the aforementioned collation with respect to the 

5 millimetric-wave objects by njn times. 

[0025] In the example shown in Fig. 2, when a millimetric-wave object Ml is fetched as 
the reference, an image object II is selected such that a millimetric-wave base pair MP1 is 
determined. When a millimetric-wave object M2 is fetched as the reference, an image object 
12 is selected such that a millimetric-wave base pairMP2 is determined. When a millimetric- 

10 wave object M3 is fetched as the reference, an image object 12 is selected such that a 

millimetric-wave base pair MP3 is determined. When a millimetric-wave object M4 is fetched 
as the reference, any one of the image objects II to 16 cannot be selected to form the 
millimetric-wave base pair because each distance between the respective image objects II to 16 
and the millimetric-wave object M4 exceeds the threshold value. When a millimetric-wave 

15 object M5 is fetched as the reference, an image object D is selected such that a millimetric- 
wave base pair MP4 is determined. 

[0026] If the image objects by the number of n j are detected by the stereo camera 3, the 
ECU 4 fetches each of the image objects one by one. The ECU 4 then collates each of 
mfllimetric-wave objects by the number of nm with respect to the fetched image object 

20 sequentially such that the millimetric-wave object closest to the fetched image object is 
selected. Likewise the collation of the image object with respect to the millimetric-wave 
object, the distance between the vehicle and the object, and the angle defined by the vehicle 
and the object are collated. In this example, fee difference between the distance from the 
fetched image object to the vehicle and the distance from the closest millimetric-wave object to 

25 the vehicle is set as a threshold value in accordance with the accuracy of the stereo camera 3 . 
Also the difference between the angle defined by the fetched image object and the traveling 
direction of the vehicle and the angle defined by the closest millimetric-wave object and the 
traveling direction of the vehicle is set as a threshold value in accordance with the accuracy of 
the stereo camera 3. If the aforementioned difference in the distance or in the angle is equal to 
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or larger than the respective threshold value, the closest millilmetric-wave object cannot be 
selected. If the closest millimetric-wave object is selected, the ECU 4 stores a pair of the 
fetched image object and the selected millimetric-wave object as an image base pair. The ECU 
4 repeats the aforementioned collation with respect to the image objects by n i times. 
5 [0027] In the example shown in Fig. 2, when the image object II is fetched as the 

reference, the millimetric-wave object Ml is selected such that an image base pair DPI is 
determined. When the image object 12 is fetched as the reference, the millimetric-wave object 
M2 is selected such that an image base pair IP2 is determined. When the image object D is 
fetched as the reference, the millimetric-wave object M5 is selected such that an image base 

1 0 pair IP4 is determined. When the image target object 14 is fetched as the reference, the 

nriUimetric-wave object M5 is selected such that an image base pair IP4 is determined. When 
the image object 15 is fetched as the reference, any one of the millimetric-wave target objects 
Ml to M5 cannot be selected to form die image base pair because each difference in the 
distance between the respective millimetric-wave objects Ml to M5 and the image object 15 

1 5 exceeds the threshold value. When the image object 16 is fetched as the reference, any one of 
the millimetric-wave target objects Ml to M5 cannot be selected to form the image base pair 
because each difference in the distance between the respective millimetric-wave objects Ml to 
M5 and the image object 16 exceeds the threshold value. 

[0028] The ECU 4 performs comparison between the millimetric-wave base pairs and 

20 the image base pans sequentially so as to select the millimetric-base pair and the image base 
pair each including the identical millimetric-wave object and the image object The ECU 4 
further sets the selected combination of the millimetric-base pair and the image base pair each 
including the identical millimetric-wave object and the image object as a fusion pair (fusion 
object). Then the information with respect to die distance and the relative speed derived from 

25 the millimetric-wave object data, and the information with respect to the angle derived from the 
image object data is set as the fusion object information. In the ECU 4, the millimetric-wave 
objects that have not been selected as the fusion object is set as independent millimetric-wave 
objects, and the image objects that have not been selected as the fusion object is set as 
independent image objects, respectively. 
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[0029] In the example shown in Fig. 2, each of the millimetric-wave base pair MP1 and 
the image base pair IP1 includes the identical miliinietric-wave object Ml and the image object 
II , thus forming a fusion pair FPL Each of the millimetric-wave base pair MP2 and the image 
base pair IP2 includes the identical millimetric-wave object M2 and the image object 12, thus 

5 forming the fusion pair FP2. Each of the millimetric-wave base pair MP4 and the image base 
pair IP3 includes the identical millimetric-wave object M5 and the image object D, thus 
forming the fusion pair FP3. The millimetric-wave base pair MP3 has no image base pair 
including the identical millimetric-wave object and the image object The image base pair IP4 
has no millimetric-wave base pair including the identical millimetric-wave object and the 

10 image object 

[0030] The collating process performed in the obstruction detection system 1 as shown in 
Fig. 1 will be described referring to a flowchart of Fig. 3. The flowchart of Fig. 3 represents 
the collating process performed by the obstruction detection system 1 as shown in Fig. 1 . 

[0031] The obstruction detection system 1 detects millimetric-wave objects (n.m objects) 
15 using the millimeter-wave radar 2 for detecting the object, and detects image objects (n.i 
objects) using the stereo camera 3 for detecting the object 

[0032] In step SI, in the obstruction detection system 1, each of the ru image objects is 
collated with each of the millimetric-wave objects fetched as a reference sequentially, and the 
image object as being closest to the reference millimetric-wave object is selected In the case 
20 where each difference in the distance and the angle with respect to the millimetric-wave object 
and the selected image object is equal to or smaller than each of the threshold values, the 
millimetric-wave base pair including the fetched millimetric-wave object and the closest image 
object is determined. 

[0033] Then in step S2, it is determined whether collation of the n.i image objects with 
25 respect to the n.m millimetric-wave objects has been completed in the obstruction detection 
system 1. Step SI is repeatedly executed until completion of the collating process. In this 
embodiment, each of the detection results of the stereo camera 3 is scanned based on the 
detection results of the millimeter-wave radar 2 such that one image object that is considered as 
being closest to the respective millimetric-wave objects is identified. 
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[0034] Then in step S3, each of the nra millimetric-wave objects is collated with each of 
the image objects fetched as a reference sequentially, and the millimetric-wave object as being 
closest to the reference image object is selected. In the case where each difference in the 
distance and the angle with respect to the image object and the selected millimetric-wave 
5 . object is equal to or smaller than each of the threshold values, the image base pair including the 
reference image object and the closest millimetric-wave object is determined 

[0035] The process proceeds to step S4 where it is determined whether collation of the n 
m millimetric-wave objects with respect to the ni image objects has been completed in the 
obstruction detection system 1. Step S3 is repeatedly executed until completion of the collating 

1 0 process. In this embodiment, each of the detection results of the millimeter-wave radar 2 is 
scanned based on the detection results of the stereo camera 3 such that the millimetric-wave 
object that is considered as being closest to the respective image objects is identified. 

[0036] In step S5, in the obstruction detection system 1, collation is performed between 
all the determined millimetric-wave base pairs and all the determined image base pairs so as to 

1 5 search a combination of the millimetric-wave base pair and the image base pair each having 

the identical millimetric object and the image object. If the aforementioned combination of die 
millimetric-wave base pair and the image base pair is searched in the obstruction detection 
system 1, such combination is determined as the fusion pair, and the fusion object information 
indicating the distance, relative speed, and the angle is set In die obstruction detection system 

20 1, two groups of the collation results, that is, the millimetric-wave base pair and the image base 
pair, derived from the bilateral collations between the millimetric-wave objects and the image 
objects are further collated. Only when each of those base pairs includes die identical 
millimetric-wave object and the image object, they are formed into the fusion objects. 

[0037] In step S6, in the obstruction detection system 1, upon determination of the fusion 

25 objects (n3 objects), each number of the independent millimetric-wave objects (nl =n_rn - n3) 
and the independent image objects (n2=ni-n3) is obtained. In this way, the obstruction 
detection system 1 determines the fusion object, independent millimetric-wave object, and 
independent image object at every detection of the millimetric-wave object and the image 
object performed by the millimeter-wave radar 2 and the stereo camera 3, respectively. 
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[0038] In the obstruction detection system 1, bilateral collation between the millimetric- 
wave objects and the image objects is performed In the case where the two kinds of such 
collation results coincide with each other, the preceding object is determined as the fusion 
object. The accuracy (fusion accuracy) in determining the coincidence of the objects detected 

5 by the image and the millimetric wave may be improved to substantially a high degree. In the 
obstruction detection system 1, one of the millimetric-wave object and the image object which 
is closest to the other object can be selected, and a combination of the millimetric-wave base 
pair and the image base pair each including the same objects can be searched through a simple 
method, thus reducing the processing load. 

10 [0039] The obstruction detection system 1 allows collation between the detection results 

based on images of the preceding vehicle and the detection results using the millimetric wave. 
This may supply the obstruction information with high reliability to various types of operation 
support systems that appropriately support the vehicle operator to drive the vehicle. 

[0040] As has been described with respect to the embodiment of the invention, it is to be 

1 5 understood that the invention is not limited to the aforementioned embodiment but may be 
embodied into various forms. 

[0041] The embodiment of the invention is applied to the obstruction detection system 
that is provided in the vehicle. It may be applicable to various types of object detection, for 
example, contact-free detection. 

20 [0042] The embodiment of the invention includes two types of detection units, that is, the 

millimeter-wave radar and the stereo camera. However, any other detection unit such as a laser 
radar may be employed Further three or more detection units may also be employed. 

[0043] In the embodiment of the invention, the position of each object is identified based 
on the distance, relative speed, and angle. However, other information such as two- 

25 dimensional coordinate system may be used for identifying the position of the respective 
objects. 

[0044] While the invention has been described with reference to exemplary embodiments 
thereof, it is to be understood that the invention is not limited to the exemplary embodiments or 
constructions. To the contrary, the invention is intended to cover various modifications and 
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equivalent arrangements. In addition, while the various elements of the exemplary 
embodiments are shown in various combinations and configurations, which are exemplary, 
other combinations and configurations, including more, less or only a single element, are also 
within the spirit and scope of the invention. 



